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We investigate many-body phase diagrams of atomic boson-fermion mixtures loaded in the two- 
dimensional optical lattice. Bosons mediate an attractive, finite-range interaction between fermions, 
leading to fermion pairing phases of different orbital symmetries. Specifically, we show that by 
properly tuning atomic and lattice parameters it is possible to create superfiuids with s-, p-, and 
d-wave pairing symmetry as well as spin and charge density wave phases. These phases and their 
stability are analysed within the mean-field approximation for systems of 40 K- 87 Rb and 40 K- 23 Na 
mixtures. For the experimentally accessible regime of parameters, superfiuids with unconventional 
fermion pairing have transition temperature around a percent of the Fermi energy. 



Mixtures of quantum degenerate atoms recently be- 
came a subject of intense studies. Examples in- 
clude recent experimental observations of instabilities in 
boson-fermion mixtures 1 *, s-wave pairing Superfluidity 2 
and condensation of molecules in fermionic mixtures 3 . 
Many other intriguing many-body effects have been pro- 
posed theoretically. They include formation of com- 
posite particles^, appearance of charge density wave 
order—, phonon-induced fermion pairing^, and polaronic 
effects—. In this Letter, we study quantum phases of 
boson-fermion mixtures (BFM) in two-dimensional (2D) 
optical lattices. We show that a number of very in- 
teresting many-body phases can be observed in such 
systems by appropriately choosing atomic and lattice 
parameters. These include charge and spin density 
wave phases (CDW/SDW) as well as superfluid states 
with unconventional pairing of fermions. Experimental 
realization of such systems should provide critical in- 
sights into understanding several important strongly cor- 
related electron systems, including quasi-2D unconven- 
tional superconductors, such as high T c cuperates^ and 
organic conductors 10 displaying d-wave superconductiv- 
ity, as well as ruthenatesi 1 « and Bechgaard saltsiS dis- 
playing p— wave superconductivity. 

In the BFM that we consider here, the fermionic atoms 
are prepared as a mixture of two hyperfine spin states. 
These two kinds of fermionic atoms interact via short- 
range repulsive interaction. Density fluctuations in a con- 
densate of bosonic atoms induce an attractive interaction 
between fermions, which is of finite-range. Competition 
between these two types of interactions results in sev- 
eral many-body phases of fermions in different parameter 
regimes. We show that these phases can be realized for 
realistic atomic systems by tuning the densities of bosons 
and fermions, lattice parameters, or interaction strength 
via Feshbach resonance. We will discuss how systems 
can be prepared and how these phases can be observed 
in experiments 12 . 

Figure ^ presents examples of mean-field analysis for 
three realistic systems that we considered (these are la- 
belled by systems A, B, and C respectively in this Letter). 
Systems A and B are chosen to correspond to 40 K- 87 Rb 
mixtures trapped in optical lattices of two different laser 



wavelengths. System A corresponds to atoms confined 
by far-off resonant Nd:YAG laser. System B corresponds 
to blue-detuned optical lattice tuned closer to the optical 
resonance of K than that of Rb. Finally system C is 40 K- 
23 Na mixture loaded in the Nd:YAG lattice. In order to 
show the main features of different competing phases we 
use fermion filling fraction and boson-fermion onsite in- 
teraction (Ubf) as a tunable parameter (fermion-fermion 
onsite interaction (Uff) is used for the inset of Fig. [He)). 
Interesting quantum phases are typically obtained by re- 
ducing the Ubf from its background value. This can be 
achieved by either relative shift of bosonic and fermionic 
latticed or by tuning the system in the vicinity of Fesh- 
bach resonanceii Other interaction strengths are based 
on the background s-wave scattering lengths^. 

Several important results presented in Fig. ^shordd be 
noted. First of all, SDW and CDW are dominant near 
half- filling region due to the nesting effects. Strong Ubf 
favors CDW (or suppresses SDW) phase as shown in (a)- 
(c), while strong Uff favors SDW (or suppresses CDW) 
phase as shown in the inset of (c). This is due to the 
induced attractive interaction (proportional to |£4/| 2 ) 
competing with the onsite repulsion between fermions. 
Second, the two density wave phases are separated by 
the superfluid phases. Whereas it corresponds mostly to 
the s-wave pairing for system A , d- and p-wave pairing 
phases dominate for the systems B and C. Physically, the 
unusual pairing mechanisms arise when the boson corre- 
lation (healing) length is comparable to lattice constant 
resulting in strong inter-site correlations. In a typical 
40 K- 87 Rb system shown in (a), such requirement cannot 
be satisfied because the 87 Rb is relatively heavy. How- 
ever, the ratio of the effective mass of bosons to fermions 
in optical lattice can be reduced by tuning the laser fre- 
quency closer to the resonance energy of 40 K atom and 
simultaneously reducing the laser intensity (system B) 
or by simply using a lighter bosonic atoms (system C). 
Fig. njb)-(c) show that both of these two systems have 
wide regime for unconventional (p- and d-wave) fermion 
pairing phases. Finally, within our meanfield analysis, a 
part of these unconventional pairing phases falls in the 
regime of collapse. However, it should be noted that 
it has been showni& that the meanfield approximation 
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FIG. 1: Phase diagrams of (a)-(b) 40 K- 87 Rb mixtures loaded 
in 2D optical lattice with wavelength A = 1060 and 765.5 nm 
(blue detuning from the D2 line of potassium by Av = 0.48 
THz) respectively, and of (c) 40 K- 23 Na mixtures with wave- 
length A = 1060 nm. We assume that 40 K are prepared in 
the hyperfme states \F,m F ) = |9/2, -9/2) and |9/2, -7/2) 
corresponding to the pseudo-spin up and down fermions. 
Bosonic atoms 87 Rb and 23 Na are assumed to be trapped 
in state rif is fermion filling fraction and Utf/Uff 

is boson- fermion/fermion- fermion onsite interaction. The in- 
plane laser intensity is chosen to be Vb = 5E r for 40 K and it is 
Vo,z = 30i? r in the z direction to form a (quasi-)2D systems. 
E r is the recoil energy of 40 K and its value is different in the 
different systems. We choose boson filling fraction m = 1 for 
systems A and B but rib — 9 for system C respectively. The 
shaded area are the regimes of collapse calculated within the 
mean field approximation. The phase diagrams for hole dop- 
ing is the same except for the minor difference in the regimes 
of collapse. Arrows indicate the interaction strength away 
from Feshbach resonance— (it is at 0.84 E r and not shown in 
(a)). The different lattice strengths for sodium and rubidium 
due to different optical transition frequencies are included in 
the calculation. 



strongly overestimates the collapse region, while corre- 
lation effects beyond the meanfield approximation often 
stabilize the uniform phases. Thus Fig. nb)-(c) demon- 
strate that there is a wide range of parameters for which 
one can observe unconventional fermion pairing in BFM 
systems in 2D optical lattice. 

The phase boundaries are determined by comparing 
the mean field T c of these many-body phases. From our 
numerical calculation, T c for the p-wave pairing phase of 



system B is about 0.2-2% of the Fermi energy (outside 
the collapse regime), and it is about 1-3% for the d-wave 
pairing phase of system C. We note that our calculations 
of transition temperatures rely on similar approximations 
as employed in analysing electron pairing in solid state 
systems with electron-phonon interactions^. 

We next detail the microscopic theory resulting in the 
above phase diagrams. When the optical lattice potential 
is strong enough, the BFM system can be described by 
the Hubbard type Hamiltoniar&iSiiii: 



n 4^ 



u bb 



(1) 



where 6k and /k iS are the annihilation operators for 
bosonic and fermionic atoms with momentum k. p^ = 



is the boson density operators, and e£ 



e k — p,b, where e k = — is the single particle energy 
with t b being the tunnelling amplitude of bosons between 
neighboring sites and 7k = 2 (cos k x + cos k y ) (lattice 
constant is set to be unit), p, b is the boson chemical 
potential. Similar notations also apply to fermions with 
superscript / and p{. = p{ T + in Eq. JIJ. U bb , U bf , 
and U/f are respectively boson-boson, boson- fermion and 
fcrmion-fcrmion onsite interaction energy, which can be 
calculated easily from the s-wave scattering length and 
the lattice potentialiS*i2iiS. fl is the system volume. For 
simplicity we do not include the global trapping potential 
and consider systems with uniform densities. 

We are interested in the low temperature regime where 
the bosonic atoms are in condensed state (ji b = Cq). 
Using Bogoliubov approximation, one can obtain an ef- 
fective fermion-phonon coupling Hamiltonian 6-20 . It is 
well-known& that the phonon field can be integrated out 
to provide an effective attractive interaction between 
fermion atoms and hence cause the fermion pairing. If 
the phonon velocity c is larger than the Fermi velocity 
Vf (i.e. in the fast phonon limit), the resulting inter- 
action between fermions is instantaneous and given by 
^ nd (k) = -V(l + £ 2 (4 - 7k )), where V = U^ f /U b is 
the strength of the phonon-induced attractive interaction 
and £ = y / t b /2n b U bb is the correlation(hcaling) length of 
the bosonic field. Such anti-adiabatic limit definitely ap- 
plies for the system C (c/vj ~ 5) but should be carefully 
examined for systems A and B (c/vf ~ 1). A common 
approach to including retardation effects is to introduce 
the energy cut-ofSii in the self-consistent gap equation 
(see Eq. @ below). Such cut-off may change a prefac- 
tor in the BCS expression for T c from the Fermi energy, 
Ef, to some characteristic bosonic frequency. For sys- 
tems A and B, bosonic frequencies are not very different 
than Ef. Hence, retardation effects are small and may 
be safely neglected. Thus in all cases, the effective inter- 
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action between fermions may be taken as 



k,s.s' 



where V^ s (k) 



U ff S s ^ s , +V ind (k). 
Following the early work of Micnas et. alm^, we 
apply the meanfield approximation to calculate the T c 
of fermion pairing phases and that of the competing 
SDW/CDW phases. For the superfhhd states, the single 
particle excitation energy, E^, has a gap at Fermi surface: 



Eh 



where the gap function, 



is determined by 



A k"' = ^E^Z(k-p) 



lanb [ ^ 
2T 



(3) 



Here £ k is fermion exchange self-energy within Hartree- 
Fock (HF) approximation and we dropped the spin index 
due to the spin symmetry. Finally the fermion chemical 
potential is determined by the known total density of 
fermions: 



= IV(ft f 
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To analyze the results of different gap symmetries, we 
consider the following ansatz for the gap functional: 



Af 



'(A s0 +A sl 7 k +A d 77 k )+^ SiS 'A p sin k x 



where 



?7k = 2 (cos k x — cos k y ). Here A s o and A s i are for the on- 
site and extended s-wave pairing phase, while A p / d is for 
the p-/d-w&ve pairing phase. The transition temperature 
T c is then numerically solved by setting A^g 



sl,d,p 



in 



Eqs. ©-0- Similar approach is used to analyze the 
SDW/CDW phases 2 ^. Away from half-filling, however, 
the CDW/SDW phases for the commensurate wavevec- 
tor, Q = (tt, 7r), may be less favorable than density 
wave phases at incommensurate wavevectorsSi, which are 
much more difficult to analyze than their commensurate 
analogues. In this Letter we only discuss SDW/CDW 
phases at (tt, 7r) for simplicity, but we note that they 
provide an accurate estimate for the regime of general 
SDW/CDW phases. 

Finally we study the stability of the resulting 
phase a 16 ! 20 by first calculating the ground state energy 
in the mean field approximation: 
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The stability condition then results from requiring 
that the compressibility (or bulk modulus, B) to 
be positive. At zero temperature it is defined by 
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FIG. 2: Typical T c of SDW phase and d-wave pairing phases 
as a function of filling fraction for system C. Uff/E r — 0.52 
and 0.6 for the lower and upper curves of SDW phase respec- 
tively. D — 4tf is half of the band width. Inset: same but 
with Uff/E r = 0.32 and 0.28 for the lower and upper curves 
of the s-wave pairing phase respectively. T c of p-wave pairing 
phase is too small and not shown for simplicity. 
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-n(dP/dn) Nf 



n 2 jd 2 £MF/dn 2 f 



where P 



— (8Emf /dQ) N is the pressure of the atomic gas and 
£mf = Emf/Q is the energy density. We note that the 
above definition of stability (i.e. B > 0) is different from 
the previous mean field theories^ - due to the additional 
gap energy of the pairing ground state. This condition 
is evaluated numerically in our calculation and the re- 
sults have been shown in Fig. ^ Similar analysis for 
CDW and SDW phases are also presented. It is impor- 
tant to emphasize that the mean field results for the on- 
set of collapse regime is overestimated due to the lack of 
correlation energjii 6 .. More sophisticated self-consistent 
calculation are generally needed for a better quantitative 
estimate. 

In Fig. [21 we show the typical T c of the SDW and d- 
wave pairing phases for system C with two different val- 
ues of Uff. For strong onsite repulsive interaction SDW 
phase is favored near the half-filling (nf — I) regime, 
while d-wave pairing become dominant only when n/ is 
further away from half-filling. In the inset we show the 
same calculation but with smaller Uff, where the s-wave 
pairing phase become dominant (see Fig. ^c)). The 
T c of s-wave pairing phase has a dip at \nf — 1| ~ 0.5 
because the onsite and the extended(inter-site) compo- 
nents of s-wave pairing are dominant near half-filling and 
away from half-filling respectively. For a typical value of 
T c - 0.01 - 0.03D, the effective BCS coupling strength 
Ascs is estimated to be 0.3 — 0.45. Therefore most of 
our results still fits reasonably well into a weak coupling 
regimes where the meanfield approximation is meaning- 
ful. 

In Fig. |3 we show the T c of d-wave pairing phase for 
system C at half-filling case as a function of £, which can 
be easily tuned by changing the density of bosonic atoms. 
We find that for different values of V . the maximum T c 
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FIG. 3: T c of d- wave pairing phase at half- filling (n/ = 1) as a 
function of the healing length, £, which can be easily changed 
by tuning the density of bosonic atoms. Curves from bottom 
to up are for V/E r = 0.53, 0.88 and 1.23 respectively, and all 
the other parameters are the same as used for system C. 

correspond to £ c ~ 0.6 (in the units of lattice constant). 
For system C, this corresponds to rib ~ 9. This obser- 
vation explains why unconventional pairing phases are 
not present in system A, which has a very short healing 
length (£ ~ 0.12) and, hence, no inter-site coupling. 

Low-temperature states of a BFM in the 2D optical 
lattice can be prepared by a process in which the sym- 
pathetic cooling of the atomic mixture is followed by an 
adiabatic cooling via increasing the lattice potential^ un- 
til the phase transition is reached. Here we describe an 
altentive method to reach the low temperature unconven- 
tional pairing phases. Consider the situation in which a 
condensate mixture of bosonic atoms and molecules is 
created. (As in recent experiments^ molecular BEC is 
composed from the two-species fermions in the repulsive 
interaction side of Feshbach resonance.) The lattice po- 
tential is then turned on and adiabatically increased such 
that the molecules are driven into a strongly localized 
state (this may not be a Mott insulator state since the 
filling fraction can be less than unity for molecules). A 
two-photon Raman puls o 24 ' 25 is then applied to dissoci- 
ate the bound molecule states into a two-atom state in 
each lattice site. Finally, the lattice potential is adia- 
batically lowered to the desired value such that an un- 



conventional pairing phase with inter-site coherence is 
generated during such "melting" process. When bosons 
with light effective mass are used, they will remian in the 
superfluid phase and mediate the attractive interaction 
between fermions during an entire procedure. We point 
out that this method can be extermely efficient since the 
initial molecule condensate ensures that all final Cooper 
pairs are close to the zero momentum states. 

The exotic many-body phases can be detected by 
various approaches. First of all, CDW order can be 
observed in a standard time-of-flight measurement of 
bosons, which produces additional peaks at wavevector, 
Q = {n, 7t). The spin density correlation and the pair cor- 
relation can be further investigated by studying the noise 
correlation in a time-of-flight measurements^ 7 .: the spec- 
trum of pairing phase will have a peak at zero momen- 
tum due to the condensation of Cooper pairs, while it will 
peak at momentum Q in the SDW phase due to nesting. 
Furthermore, one can use Bragg scattering spectroscopy 
to probe the gap symmetry of the fermion pairing as 
proposed in Ref. |l9j |. For example, for d-wave pairing 
phase a zero energy excitation should be observable when 
the momentum transfer of the two scattering photons is 
tuned to match any two of the nodal points on the Fermi 
surface. One can also use rf-spectroscopy to measure the 
binding energy of fermion-pairing^ and the photoassoci- 
ation method to measure the superfluid fraction of the 
fermion pairs 26 . These techniques are presently used by 
a number of experimental groups. 

In summary we investigated the many-body phase di- 
agrams of a boson-fermion mixture in 2D optical lattice. 
For a realistic 40 K- 23 Na or 40 K- 87 Rb system, a nature 
of superfluidity can be controlled by appropriate tuning 
of the atomic interaction interaction strength and opti- 
cal lattice parameters. Our results indicate that experi- 
mental studies of boson-fermion mixtures of cold atoms 
could have important implications for understanding the 
physics of unconventional superconducting materials. 
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